The present invention is generally directed to subjective listening tests. More specifically, the present invention is directed to a method and system for administering subjective listening tests to remote users.
For service providers providing voice communication services (telephone, cellular, Voice over Internet Protocol (VOIP), etc.), it is necessary to be able to assess the perceived quality of technologies and services by customers. Subjective listening tests are typically used for this purpose. A subjective listening test involves presenting test samples to customers and gathering the customers' opinions of the quality of the test samples. The test samples are typically made up of audio recordings that capture various conditions of interest, such as circuit noise or packet loss, or have been made over competing networks or new vendor equipment. Subjective listening tests can be used to perform vendor equipment evaluation and certification, competitive assessment, service troubleshooting, network and service planning, and to test emerging technologies, such as VoIP.
A common type of subjective listening test is a Mean Opinion Score (MOS) listening test. MOS listening tests provide a numerical measure of the perceived quality of voice communications delivered using a particular service or technology. In MOS listening tests, customers (“users”) listen to multiple recordings of telephone network connections. After each recording is played, the users rate how the recording sounds to them. The users rate the recordings using a rating scale of “5”, “4”, “3”, “2”, and “1”, meaning excellent, good, fair, poor, and bad, respectively. For each recording, the ratings for all of the users are averaged to determine the MOS for the recording. The various recordings can be used to test different systems (i.e., VoIP, cellular, PSTN), equipment, software, or different conditions (i.e., packet loss circuit noise, echo noise) in a system.
Conventionally, MOS listening tests are performed in a lab setting. In order to perform MOS listening tests, a lab must be equipped with expensive, specialized equipment that is used to simulate a typical telephone and access network of users. This lab must have equipment to play the sample recordings to users, collect user ratings, and record the data in a useable format. For example, a computer system plays the recordings, which are sent to an attenuator to control the level of the recordings. The recordings are then sent to a splitter which sends the recordings to the different users. The users listen to the recordings over hand sets that are calibrated so each user hears the same thing. Accordingly, each user hears the same recording at the same level at the same time. The users enter their ratings using a touch screen or the like, and the ratings are saved and averaged on a computer system.
Due to the cost of purchasing and calibrating the equipment, it is expensive to set up and maintain a lab for MOS testing. Furthermore, users must be paid not only a fee for their services in completing the MOS testing, but also a premium for their travel to the lab site. Additionally, scheduling users to come to a lab for MOS testing and running an MOS listening test for a group of users is time consuming.
In addition to the MOS listening test, a variety of other subjective listening tests can be used to assess user opinion of voice communication technologies and services. For example, a paired-comparison test directly compares two audio samples, a Modified Rhyme Test (MRT) measures user opinion of intelligibility of audio samples, and a Degradation Category Rating (DCR) test rates a user annoyance level of distortion or other impairments. Similar to the MOS listening test, other subjective listening tests are conventionally performed in a lab setting are can be costly and inefficient to administer.